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The invention is described in the following statement: 



A SCALABLE COMPUTER SYSTFM 



This invention relates to computer systems '■■ 
. exclusively, to a computer system which is required to have a large numlier.of 
5 computer processors for use m large scale 

Recent devdopments in teleconmiu^^ 
primarily involving the field of photonics, are resulting in a rapid expansion of 
bandwidths available for communication^^ , 
growing at a rate involving, roughly, a factbr of two every two years, and it is 
10 anticipated that communication bandwidths may increase by at least three prders of 
magnitude over the next ten years. 

In order to match this rapid growth in the level of teleconmiunications, 
. equivalent computing power is required. It is therefore desirable to design a ; 
computer based systena arclutecture that is m^^ The scalability is 

15 essentially driven by the number of independent users^ as in mobile phonie devices, 
ratheir than the complexity or size of an indi^^ 

Hitherto, there has been no known solution to the problem of designing a 
massively scakble architecture for the intelligent network (IN) domain. There have 
been applications, such as enciyption, that have been nui over lafgei iiumbers of 
20 independent Internet-connected system?, based oh the concej^t fliat the Internet itself 
is an extremely large computer system. However, the ftitemet is a hierarchicai 
system and such applications do not address the problem of designing a massively 
scalable computer system for use in the intelligent networks. Consistent system 
architectures that are known in that area are many orders of magnitude leiss than 
25 what will be required in the near future. 

It is therefore desirable to jwovide a massively scalable compiuter system 
including a large number of processors in which each processor can communicate 
effectively with other processors without regard to their locations. 

According to one aspect of the invention, there is provided a large scale 
30 computer system comprising a multiplicity of nodes, each node having a plurality of 
interconnected processors, said nodes being arranged in a network with _: 
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neighbouring sets of nodes of the network fonning neighbourhoods of fully . 
interconnected nodes, wherein random links are provided between nodes of 
different neighbourhoods ini the network whereby each processor of tiie system can 
communicate effectively with other processors regardless of their location in the 
5 network without fiill connectivity in the network. 

The present invention utihses a message based approach.and a "small- world" 
network architecture in which a relatively small number of random cross-links of 
nodes or vertices in a netwdrk can resUlt^iii small characteristic path lengths for the 
transfer of messages between nodes or vertices in the network regardless of their 
10 location- 

In very large networks with only local connections most verticesf or nodes in 
the network are separated by many links. In regular lattice or ring structures with 
local connections between vertices or nodes, the characteristic or mean distance of 
the path length (L) between two vertices or nodes grows approximately linearly 

15 with the size of the network. On the other hand, in a network with only random , 
connections between vertices or nodes, the characteristic path length (L) grows only 
logarithmically with the number of vertices or nodes. However, such a network is a 
poorly clustered environment in which a sparse number of random connections can 
result in some vertices or nodes tiot being connected to o^ 

20 A small world network is one which falls between a regular network and a 

coiflpletely random network in fiiat vertices or nodes in local neighbourhoods or 
clusters are interconnected to each other, and a relatively small number of random 
links or connections are provided between nodes of different neighbourhoods of the 
netwoik. 

25 Structural properties of small-world networks can be defined mathematically 

in terms of their characteristic path length L(p) and clustering coefficient C(p). L(p) 
measures the typical separation between two vertices in a network (a global 
property) and C(p) measures the clustering or connections in a typical 
neighbourhood (a local property), where p, is a probability factor. In a regular 

30 network having n nodes and k edges or liiiks per nodes, p = 0, L (0) grows linearly 
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with n, the number of nodes in the network^ and C(0) depends on the specific . 
geometry or 'wiring' of the network. 

hi a completely random network with only random connections, p = 1 , 
. L-random grows only logarithmically with n, whereas C-:r£uidom ±k/n«l. 
5 Examples of regular, small- world and random networks are shown in Figure 1; 
Studies of natural, social and man-made networks, sudi as the neural 
network of the worm C. eleganis, a collaboration of iSlm actors and the power grid 
of tiie western UlS. A. (Cdllectiye dynamics of "small-wbrld networks" by Duncan 
J. Watts & Steven H. Strogat^, Nature, Vol 393, pp 440-442) have shbwn Ihat such 
10 networks are "smal^world" networks and that there is a broad interval of p over 
which L(p) is almost small as L-random, yet C(p) is much greater than C-random. 
Iliis is shown in the graph of Figure 2. 

Watts attd Strogatz'have thus deni 

15 comi^ctions are sufiBcient to turn a regular 

reduce the path length of the number of links for effective communication between 
nodes or vertices drastically. Hanspeter Herzel in his article entitled "How to 
Quantify Small-World Networks" (Fractals, Vol 6, No, 4 (1998) 301-303) has also 
derived a simple fonnula for the iriean eoniiectiyity^ 

20 ring network wiifli k edges of links per node, in which ik -. fiactioh of p connections arei 
re-wired randomly: 

-.•k^-k.p:".-^ 

When k > 1, the characteristic path length L^) may be expressed as . 
follows: 

25 ln(n) ln(ri) 



L(p)^ ln(k'^) ln(k) + ln(p) 

Such a logarithmic curve approximates the curve L(p)/L(0) in Figure 2. 

In the present invention, the iiuinber of nodes in the network, the number of 
30 nodes in the neighbourhood, the number of nodes per neighbourhood and the 
nunibei of cioss-links iu Uie iielwork may eauli be varied for different applica t ions. 
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provided that the netwbrk functions as a small-world network with largie clustering 
C(p) and small average path lengths L(p). 

The crossrlinks in the smail-world^ 
chosen completely a[t random. Altematively, a pseudo-random selection process 
5 may be used to select the crossrlinks between neighbourhoods to convert a regular 
network into a highly clustered small-world network with a relatively small average 
pathlength. 

In one preferred embodiment of th^e invention a mem cp^ 
substantially in the range from about 1 .5 to about 2.0, and preferably aboiit 1 ;6, may 

iO be achieved by appropriate choice of the number of nodes per neighbourhood, the 
connectivity of each neighbourhood and the nxmiber of cross-links relative to the 
total number of nodes in the network. By way of example, in a computer system 
having 50 neighbourhoods of 10 nodes arranged in a ring network with about 50 
cross-links between neighbourhoods, each node is connected to 9 other nodes in its 

15 neighbourhood (k= 9) and the probability factor p^= 
' .k"^= 9x100 =1.8. 

: " 500: • • 

The number of intercbnnected processors in each node of the computer 
systenimay dso vary for diffCTdat applicati With the development of 
. 20 photonics, it is envisaged tiiat up to about 256 nodes, each containing up to eight 
processors per node may be folly connected together by optical fibres and a high 
speed switch to form a single neighboiurhood in a small-world network. With a 
large number of such neighbourhoods in the small-world network connected by a 
relatively small nuniber of cross-links, it is possible to achieve ai massively scd 
25 computer system with a very large mmiber of total processors m the system which 
are able to commimicate with each other in an effective m£^ 

A preferred embodiment of the present invention will now be described, by 
way of example only, with reference to the accompanying drawings in which: - 

Figure 1 is a diagram showing the transition from a regular ring network to a 
30 random network via a "small- world" network; 
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Figure 2 is a graph showing variations in clustering and average path length 
with increasing rimdpmness in a ring network. 

Figure 3 is a diagram of a single node computer system which has limited 
scalability; 

5 Figure 4 is ia diagram of a spliindde cdm^ 

Figure 5 is a diagram of a four node computer system-having four processors 
per node; 

Figure 6 is a diagram of a small-world architecture for a computer system- in 
accordance with the invehtipn, 

10 The computer system of Figure 3 is b^ised on an Erlang/Open Telecorai 

Platform (OTP) running on a single node. The computer system 10 includes 
hardware 12, an operating system 14, a display 16 and a keyboard 18 and a suite of 
programs 20 which include application prbgranis 22 (eg. in programming 
languages, Erlang and C), sourced programs 24, run-time programs 26, a library 28, 

15 and a database 30. The system may biei linked to an extemal database 32 if required. 

The single node provides very good system development facilities j including 
an Erlang real time environment or intCTpretiye enviroiraient. However, this is 
achieved at the expense of potential perfonnande owing to the inteipreter/opCTatirig 
system layers. 

20 The single node computer syst^ of Fig^ 

nodes by an asynchronous transfer node (ATM) switch, such as thei AXD-301 
switch with satisfactory performance. However, this switch has a scalability of 1 :30 
which is orders of magnitude less than that which is required for ultra high 
communication bandwidths. 

25 Ilefening to Figure 4, there is shown a split node computer syrt^ 

an OTP node 34 is split into two closely couple nodes: a COTS (commodity of the 
shelO system 40 and a multi-processor (MP) Erlang Engine 50. The COTS system 
is essentially the base system and may comprise of a UNIX operating platform 41, 
application programs (eg. in C, C++, Java and Erlang) 42, a disc drive 43, graphics 

30 44, an Internet rnodem 45, an Internet interface (TCP/IP) 46 and ^a^ 
interf a ce (I/O) 4 7 for communicating with the Erlang Engine 50. — ' — — - 
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The Erlang Engine 50 is a shared inempry MP system running Ericsson core 
software 52 and possible outsourced software 54 in Erlang on top of an optimised 
message passing kemel (56) such as QNX. The Erlang Engine 50 also has an I/O 
interface 58 for communicating wifli the COTS system 40. One processor of the 
5, MP set can be devoted to moiiiitoring software, the remainder tp functional 
processing. ;^ 

The split node OTP system 34 of Figure 4 niay form part of a network which 
includes a plurality of regional processors (RP) 61 and support^ p^^ 
for operators. The regional arid support processors 61, 62 are connected to ce^ 
10 processors CP A 63 and CP B 64 and to each other by a High speed RP bus 65, and 
the MP Erlang Engine 50 includes a high speed interface 59 for communicating 
witfi the central processors CP A 63 and CP B 64. 

The spht node OTP syistem 34 can be linked to other computer systems by a 
switch 70 such as an AXE-10. For this purpose an AXE programming system 
15 (APS) 72 may be provided. Iri a telecommunication application t^^ 

shown to the AXE-10 may be impleniented as a high speed Ethernet, primarily due 
to the availability of the Ethernet PLEX (Programming 

blocks existing on AXE- 1 0. The Erlang to PLEX interface has been demonstrated 
in two modes, firstiy with the AXE-10 controlling the Unks, as in a call forwarding 
20 application. The second mode is witfi OTP in control, with an application such as 
remote changes to AXE-10 tariff tables. 

The limit for scalability for the Erlang Engine is for a majdmum of eight 
processors, given that it is a shared memory environment. The eight processors^ 
together with a demonstrated 5 x speed up from the move to compiled code, plus 2 
25 X moving from Unix to QNX gives a scale-up of 80 from the base system. 
Referring to Figure 5, there is shown a foiir node/fow 
computer system which demonstrates that it is possible to link up several 
Erlang Engines of the type shown in Figure 2 through a high speed blocking switch. 
Each node of the computer system of Figure 3 is an Erlang Engine 150 with four 
30 processors 152 running ona QNX kernel 156. Each Erlang Engine 150 may 
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A network connected in this manner would normally be termed a regular 
network in which messages from one node, eg Ai, must pass through a relatively 
large number of links to reach a node at the opposite side of the network, eg C4. 
A smali-worljd network such as shown^^^m 
5 regular network in that a relatively small number of randoni cross-links are 

provided between neighbourhoods of nodes. In the example-ef Figure 4, there are 
three cross-links CLi, CL2 and CL3. Cross link CLi directly connects nodes A2 and 
Ci with each other, cross link CL2 directiy connects nodes Brand E3 with each 
other and crossrlinkCLa dii^ It will 

10 _ be apparent from Fijgure 6 that there is a marked increase in connectivity with only a 
relatively small number of cross-links between neighbourhoods of nodes in the ring 
network. For instance, a message to be sent from node Ai to C2 can pass along an 
edge link EL to node A2, along cross-link CLi to Q and then along an edge Imk to 
C2, rather than along four loop links to C i and an edge jink EL to C2. Also, a 

15 message from B4 to E2 can pass along a loop link to B2, along cross-link CL2 to E4 
and then along another loop link LL to E2, rather than along five loop links. 

It will be seen from Figure 6 that a small-world network is an architecture 
that can be. used to Imk together a relatively large number of computer processors 
while retaining effective connectivity between the p^ 

20 neighbiburhood is a four node/four processors per node system similar to that of 
Figure 5, a total of eighty processors can be effectively linked together by the 20 
node ring network of Figure 6 wi 

the small- world network of Figure 6 is, however, only a relatively sfmall 
scale eXaiiiple of such a network. Within the scope ofthe present invention it is 

25 contemplated that at least 500 neighbourhoods of nodes each having up to 256 fiiUy 
connected Erlang Engines per node with 8 processors per Etlang Engine could be 
linked together in a small-world network to achieve a total system of over one 
million processors, with the small- world network architecture providing effective 
connectivity between the nodes with only a relatively small number of cross-links, 

30 say 50, between the neighbourhoods. 
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For example, the total nximber of individual Erlaiig processes running on 
such a system will be of the ordCT of 256,000^000 assuming 2000 active processes 
■ per Erlmg Engine node. This g;ives around 25.6 million lines of code which is of 
the order of magnitudie already eriyisag^^^^ 
5 It is envisaged that a massively scalable computer system in accordance with 

the invention has widespread applications. In the teleconmiunieations fieldj the 
scalability could cover application^ sucli as mobile telephone services for 
istbckbroking/betting and oA^ 

it will be appreciated that various modifications and alterations may be made 
10 to the present invention as; described above without departing firom the scppe and 
spirit of the present invention. For instance, as mentioned above, tjie size of the 
network, the neighbourhoods in the nietwork, the number of nodes per 
neighbourhood, the number of cross-links between neighbourhoods and the number 
of processors per node may be varied for different applicat^^ 
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DATED: 30 June 1999 

CARTER SMITH & BEADLE 
latent Attorneys for the Applicant: 

TELEFONAKTIEBOLAGETLM ERICSSON 
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